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Abstract 



We find a closed formula for the triple integral on spheres in 
' M^" X M^" X R^" whose kernel is given by powers of the standard 

symplectic form. This gives a new proof to the Bernstein-Reznikov 
. integral formula in the n = 1 case. Our method also applies for linear 

and conformal structures. 

B-. 

^ ; 1 Triple product integral formula 

> : 

. We consider the symplectic form [ , ] on M^" = M" © M" given by 

OO 

CM : [{x, 0, (y, v)] ■■= -{x, v) + (y, 0- (i-i) 

O ' In this paper we prove a closed formula for the following triple integral: 

o^ : 

^ ' Theorem 1.1. Let da he the Euclidean measure on the sphere S'^'"'^^ . Then, 

r ' r ot — n p — n 7 — n 

X; / ^ |[Z,X]| 2 2 d(r{X)d(r(Y)d(r{Z) 

-r ( 2~n+a \-rf 2-n+/3 \j^f 2-?i+7 f 5+n ^ 

= f2;r"~^) '^^^ l^^M [^—)^ [—) 

^ ^ r(n)r(i^)r(^)r(^) 

Here, a = Ai - A2 - A3, /3 = -Ai + A2 - A3, 7 = -Ai - A2 + A3, 5 = 
-Ai - A2 - A3 = a + /3 + 7. 
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The integral converges absolutely if and only if (Ai,A2,A3) G lies in 
the following non-empty open region (see Proposition I6.8p defined by: 

Rea>n-2, Re/3 >n-2, Rej >n-2 {n>2), 
Rea > n - 2, Re/3 > n - 2, Re7 > n - 2, Re(5 > -1 (n = 1). 

The integral under consideration extends as a meromorphic function of Ai, A2 
and A3 ([31 Theorem 2], fH], see also [Z]). A special case {n = 1) of Theorem 
11.11 was previously established by J. Bernstein and A. Reznikov jS]. 

The strategy of our proof is to interpret the triple product integral as 
the trace of a certain integral operator, for which we will find an explicit 
formula of eigenvalues and their multiplicities. For this, our approach uses 
the Fourier transform in the ambient space, appeals to the classical Bochner 
identity, and finally reduces it to the special value of the hypergeometric 
function 5F4. It gives a new proof even when n = 1. 

Sections [2] and [3] are devoted to the proof of Theorem II. 1[ In Section HJ 
we discuss analogous integrals of the triple product kernels involving \x — y\'^ 
or \{x,y)\'^ instead of . 

The underlying symmetries for Theorem 11.11 are given by the symplectic 
group Sp{n, M) of any rank n, whereas those of Theorem 14.21 correspond to 
the rank one group SOolm + 1, 1). Even in rank one case, our methods give 
a new and simple proof of the original results due to Deitmar [6] for the case 
\x — (see Theorem 14.21) . Section [5] highlights some general perspectives 
from representation theoretic point of view. 

In Sections [2] and E] we have made an effort, following questions of the 
referee, to explain the role of meromorphic families of homogeneous distri- 
butions and the precise condition for the absolute convergence of the triple 
integral, respectively. 

Notations: N = {0, 1, 2, . . .}, M+ = {x G M : x> 0}. 

2 Eigenvalues of integral transforms 7^ 

We introduce a family of linear operators that depend meromorphically on 
/i G C by 

defined by 

i%f)iv)--= [ fiuj)\[uj,r^]\-'-''daiuj). (2.1) 
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The integral f l2.ip converges absolutely if Re/x < —n + 1, and has a 
meromorphic continuation for /i G C. If /i is real and sufficiently negative (e.g. 
/i < — n), then the kernel function /^(a;, r/) := |[a;,?7]|~'^~"' is square integrable 
on 5*^""^ X S'^"'"^ and K{uj,ri) = Kijj.uj), and consequently, 7^ becomes 
a self-adjoint, Hilbert-Schmidt operator on L^(S'^"~^). In this section, we 
determine all the eigenvalues of 7^ and the corresponding eigenspaces (see 
Theorem 12. ip . 

2.1 Harmonic polynomials on M^" and C"^ 

First, let us remind the classic theory of spherical harmonics on real and 
complex vector spaces. 

For A; G N, we denote by 'H'^(M^) the vector space consisting of homoge- 



neous polynomials p{x 




In the polar coordinates x = ru; (r > 0, u; G "'^), we have 




where A5iv-i denotes the Laplace-Beltrami operator on the unit sphere en- 
dowed with the standard Riemannian metric. In light that r-^p = kp for a 
homogeneous function p of degree k, we see that the restriction pls^-i for 
p G 'H''{R^) belongs to the following eigenspace of A^jv-i: 

Vk{S^-^) := G C°°{S^-^) : AgN-iy^ = -k{k + N- 2)(^}. (2.2) 

Since homogeneous functions on are determined uniquely by the re- 
striction to S^~^, we get an injective map 

?^^(M^) ^ \4(5^-i), p^p\sN-i 

for each /c G N. This map is also surjective (see PI Introduction, Theorem 
3.1] for example), and we shall identify '^^(M^) with Vk{S^~^). Thus, we 
regard the following algebraic direct sum 

oo 

n{R^) := 0?/'=(M^) 

k=0 
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dense subspace of C°°{S^ ^). 
Analogously, we can define the space of harmonic polynomials on C". For 
a, /3 G M, we denote by ?^"'^(C"') the vector space consisting of polynomials 
p{Z, Z) on C" subject to the following two conditions: 

(1) p{Z, Z) is homogeneous of degree a'm.Z= (zi, . . . , and of degree (i 
in Z = {zi, Zn). 

Then, T-t°''^{C^) is a finite dimensional vector space. It is non-zero except for 
the case where n = 1 and a, (3 > 1. 

By definition, we have a natural linear isomorphism: 

7/fc(M2n^ ^ 7/"./3(c"). (2.3) 

We shall see that ?^"'^(C"') is an eigenspace of the operator 7^ for any 
fi and for every a and /3. To be more precise, we introduce a meromorphic 
function of fi by 

{0 {k : odd), 

.r(i^)r(^) ^ ^ (2.4) 
27r" 2 f — ^ (fc : even). 

We shall use the notation Afc(/i,C"') when we emphasize the ambient 
space C" (see (gSD). 



Theorem 2.1. For a,(3 eN, 



= (-l)^A„+,(/i)id. 



The rest of this section is devoted to the proof of Theorem 12.11 
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2.2 Preliminary results on homogeneous distributions 

In this section we collect some basic concepts and results on distributions 
in a way that we shall use later. See [71 Chapters 1 and 2], and also fl2\ 
Appendix]. 

A distribution F^, depending on a complex parameter u is defined to be 
meromorphic if for every test function if, {F^,, if) is a meromorphic function 
in z/. We say F^ has a pole at z/ = i/q if {F^i f) has a pole at u = uq for some 
(f. Then, taking its residue at z/q, we get a new distribution 

if I-)- res {F^, ip), 

u=uo 

which we denote by res F^. 

u=uo 

Suppose F is a distribution defined in a conic open subset in R^. We say 
F is homogeneous of degree A if 

^ d 

ll^^d^F = ^F (2-5) 

j=i 3 

in the sense of distributions, or equivalently, (F, ip (^■)) = a^^'^ {F, ip) for any 
test function ip and a > 0. 

Globally defined homogeneous distributions on are determined by 
their restrictions to \ {0} for generic degree: 

Lemma 2.2. Suppose f is a distribution on which is homogeneous of 
degree X. If /|Miv\{o} = and X ^ {-N, -N - 1, -N - 2, . . .} , then / = 
as distribution on M^. 

Proof. By the general structural theory on distributions, if supp / C {0} 
then / must be a finite linear combination of the Dirac delta function 6{x) 
and its derivatives. On the other hand, the degree of the delta function and 
its derivatives is one of —N, —N — 1, —N — 2, ... . By our assumption on /, 
this does not happen. Hence, we conclude / = as distribution on M^. □ 

For a given function p e C°°{S^~^), we define its extension into a homo- 
geneous function of degree A by 

Pxircu) := r^p{u), (r > 0, cu E S^'^). 

We regard the locally integrable functions as distributions by multiplying the 
Lebesgue measure. 
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Lemma 2.3. Let p e C°°(S'^-^), then 

1) px is locally integrable on if lie X > —N. 

2) Px extends to a tempered distribution which depends meromorphically on 
A G C. Its poles are simple and contained in the set {— A^, —N — 1, . . . }. 

3) The distribution px is homogeneous of degree A in the sense of if A 
is not a pole. 

Proof. 1) Clear from the formula of the Lebesgue measure dx = r^~^drda{u) 
in the polar coordinates x = ru {r > 0, oj E S^~^). 

2) Take a test function ip G iS(M^). Suppose first ReA > ~N. Then, we 
can decompose {px, (p) as 



{p\,(p) = / px{x)(p{x)dx + / px{x)(p{x)dx. 

J\x\<l J\x\>l 

The second term extends holomorphically in the entire complex plane. Let 
us prove that the first term extends meromorphically in C. For this, we fix 
G N, and consider the Taylor expansion of 

^-^ a! 

|a|<fc 

where a = (ai, ... ,aj\f) is a multi-index, x"" = x"^ ■ ■ ■ x"^^ , |a| = + - ■ ■+aAr, 
and (pk{x) = 0{\x\'''^^). Accordingly, we have 

Px{x)ip{x)dx 

\x\<l 

y [ p{uj)uj''da{uj) r r^+H+^-irfr + f px{x)^k{x)dx 

X , I I , Ar I / p{uj)u''da{u) + / px^k{x)dx. 

^A+|a|+A^ a\ JgN-l JyA<l 



The last term extends holomorphically to the open set {A G C : ReA > 
—N — k}. Since k is arbitrary we see that {px-,^) extends meromorphically 
to the entire complex plane, and all its poles are simple and contained in the 
set {— A^, — A^ — 1, — A^ — 2, . . . }. Thus, the second statement is proved. 
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3) The differential equation ^^2;^— — p\{x) = \px{x) holds in the sense 

of distributions for Re A ^ 0. This equation extends to all complex A except 
for poles because the distribution p\ depends meromorphically on A. □ 

Example 2.4 (/c = case). Fork = 0, 7{'^(M^) is one- dimensional, spanned 
by the constant function 1 . We denote by the corresponding homogeneous 
distribution Ix- 

As we saw in the proof of Lemma \2.3\ the distribution r'^ has a simple 
pole at X = —N and its residue is given by 

^res^r' = yo\{S''-')6{x) = ^^S{x). (2.6) 

Example 2.5 (A^ = 1 case). In the one dimensional case, S^~^ consists of 
two points, 1 and —1, and consequently, the homogeneous distribution p\ is 
determined by the values p\{l) and p\{—l). From this viewpoint, we give a 
list of classical homogeneous distributions on M. 



Pa 


x\ 


x^ 


\x\^ 


xp^sgna; {x + i^Y 


{x-i^Y 


p(l) 
p(-l) 


1 




1 


1 

1 


1 1 

-1 e''^^ 


1 

g — iTrA 



Table 2.5.1: Homogeneous distributions on R 

The notation {x ± iQ)^ indicates that these distributions are obtained as the 
boundary values of holomorphic functions in the upper (or lower) half plane. 
For Re A > -I, 

lim(x ± ie)^ = (x ± zO)^ 

holds both in the ordinary sense and in distribution sense. The distribu- 
tions {x ± i{])^ extend holomorphically to all complex \, whereas the poles 
of Xj_, \x\^, |x|'^sgnx are located at { — 1, — 2, — 3, . . . }, {— 1, — 3, — 5, . . . }, 
{—2, —4, —6, . . . }, respectively. 
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For A 7^ —1, —2, . . . , any two in Table \275J\ form a basis in the space of 
homogeneous distributions of degree A. For example, by a simple basis change 
one gets: 

{x - iO)^ = e"*^(cos -Ajxl'^ + i sin -A|x|'^ sgnx). (2.7) 

2.3 Application of the Bochner identity 

Let ( , ) be the standard inner product on M^. We consider the Fourier 
transform J-" = J^^^^n on normahzed by 



and we extend J-" to the space iS'(M^) of tempered distributions. 

If / G iS'(M^) is homogeneous of degree A, then its Fourier transform J-'f 
is homogeneous of degree —A — A^. 



Example 2.6 (A^ = 1 case 

-f (A+l)p/\ , i\ 

1) -^(4)fa)= (2.)>ii '^-'«> 



2) J^{\x\^){y) = , \ ^ \ \y\-^-\ and 

J^(|x|^sgnx)(y) = -^^^r^^^\y\~^~^sgny. 

These formulas may be found for instance in ^ Chapter II, §2.3], how- 
ever, we shall give a brief proof because its intermediate step (e.g. (12. Sp 
below) will be used later (see the proof of Proposition I2.13p . 

Proof of Example \2. 6[ 1) Suppose Re A > —1. Then is locally integrable 
on M, and we have 

lime-2""^xi = xi 

both in the ordinary sense and in the sense of distributions. Then, by 
Cauchy's integral formula and by the definition of the Gamma function, we 

get 

^(e---xi)(y) = \ y - ^^r'-\ (2-8) 
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for e > 0. Taking the limit as e — )■ we get the desired identity for Re A > —1. 

By the meromorphic continuation on A, the first statement is proved. 

2) Similarly to 1), we can obtain a closed formula for J-'{x'^){y). Then the 
second statement follows readily from the base change matrix for the three 
bases sgn x}, and {{x + iO)^ , {x — iO)'^} for homogeneous 

distributions on M. (We also use the duplication formula of the Gamma 
function.) □ 

We are ready to state the main result of this subsection. Let us define 
the following meromorphic function of A by 

-p(k±X±N_\ 

Lemma 2.7. For any p G Ti'^iM.^), we have the following identity 

J^px = BN{X,k)p^x-N (2.9) 

as distributions on that depend meromorphically on A. 

Example 2.8. Since ( \2.9\\ is an identity for meromorphic distributions, we 
can pass to the limit, or compute residues at special values whenever it makes 
sense. For instance, let k = 0. Then, by (12.61) . the special value of ( \2.9\\ at 
A = yields 

.F(l) = limfijv(A,0)r-"-^ = 5(y). 

A — >-0 

In view of the identity i?Ar(A, 0)i?Ar(— A — A^, 0) = 1, the residue of (12.91) at 
A = —N yields 

J^{5{x)) = 1. 

This, of course, is in agreement with the inversion formula for the Fourier 
transform. 

Proof of Lemma \2. 7\ For N = 1, k equals either or 1, and correspondingly, 
P\ is a scalar multiple of or |x|'^sgnx, respectively. Hence, Lemma [2.71 
in the case = 1 is equivalent to Example 12.61 2). 

Let us prove (12. 9p for > 2 as the identity of distributions on R^. We 
shall first prove the identity (12. 9p on \ {0} in the non-empty domain: 

-A^<ReA< -^(iV + 1). (2.10) 
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Since the both sides of fl2.9p are homogeneous distributions of the same 
degree, this will imply that the identity fl2.9p holds on by Lemma 12.21 
Further, since the both sides of (12. 9p depend meromorphically on A by Lemma 
12. 3[ the identity (12. 9p holds for all A in the sense of distributions that depend 
meromorphically on A. 

The rest of this proof is devoted to show (12. 9p on M.^ \ {0} in the domain 
(12.100 . For this, it is sufficient to prove that 

{J'px,gq) = BN{X,k){p_x-N,9q), 

for any compactly supported function g G C^(R+) and any q G 7/'(M^) 
(/ G N) because the linear spans of such functions form a dense subspace in 
C^(R \ {0}). Here, gq stands for a function on \ {0} defined by 



N~l■^ 



{gq){sri) = g{s)q{ri) {s>0,r] E S 

By definition of the Fourier transform on 5' (M^), {J^Px,gq) = {px,J^{g<l))- 
Hence, what we need to prove is 

{px,J'igq)) = B^{\,k){p_x-N,gq)- (2.11) 

We note that both px and p-x-N are locally integrable functions on under 
the assumption (I2.10p . To calculate the left-hand side of (12.110 . we use the 
Bochner identity for q G 'H'(M^): 

q{uj)e~"'^'''''^doj = {27i)^r^u'-^ Ji^N_^{u)q{r]), 

where Jfiii') denotes the Bessel function of the first kind. Then, we get the 
following formula after a change of variables x = 27Trs: 



F{gq){ruj) = 2711 r q{uj) j s ^ g{s)Ji^N_-^{2TTrs)ds. 
Hence, we have 



oo 

AT 



{Px,J'{gq))= / / I{r,s)ds]p{u)q{uj)da{uj)dr, (2.12) 
Jo Js^-^ vo ^ 

where we set 

/(r, s) := 27rz"V^+^s^5'(s) J;^]v_-^(27rrs). 
At this point, we prepare the following: 
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Claim 2.9. Assume that A satisfies fl2.10p and that g is compactly supported 



1) /(r,s) G X M+,rfrc/s). 

2) / J(r,s)ds = 5^(A,/)(7(s)s 

^0 



-A-l 



Proof of Claim \2.S\ 1) Since tlie support of g is away from and oo, it follows 
from the asymptotic behaviour of the Bessel function J^{z) as z — )• and 
z — i- oo that there exists a constant c > such that 



|/fr,s)| < 



c r^t'^+^+'-i as r -> 0, 
c T-Rf^-^+IC^-i) as r — )■ oo. 



By the assumption — < ReA < —\{N + 1), we conclude I{r,s) is an 
integrable function on IR+ x M_,_. 

2) This is a direct consequence of the following classical formula of the 
Hankel transform |8l 6.561.14] 



Jo -L [ 2~ 



for Re(/i + z/) > — 1 and Re/x < — |. □ 

Returning to the proof of Lemma [221 we can now apply Fubini's theorem 
for the right-hand side of fl2.12p to get 

{Px,J^{9Q))=U P{^)<l{^)dcr{uj)^ U I{r,s)ds^dr 



oo 

A-l, 



= Bn{X,1) / p{Lo)q{Lo)da{uj) / g{s)s~^~^ds. 

We recall that p G 'H'^(M'^) and q G T-L''{W^), therefore the first factor is 
non-zero only if A; = /. Then the right-hand side equals 

= BN{\,k){p^x-N,gq)- 

Hence (12. lip is proved in the non-empty open domain of A satisfying the 
inequality f l2.10p . Therefore, the proof of Lemma [2.71 is completed. □ 
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2.4 Fourier transform of homogeneous functions 

We consider the restriction of the Fourier transform T on to the space 
of homogeneous functions. For /i G C, we set 

= K^(M^) (2.13) 
:= {/ G C°°(M^ \ {0}) : /(tX) = |tr'^-T/(X) for any t G M \ {0}}. 

Then, may be regarded as a subspace of the space S'{R^) of tempered 
distributions for yU 7^ y + 2, . . . . We note that / G is determined by 
the restriction /l^jv-i, and thus can be identified with the space of smooth 
even functions on S^~^. In this subsection, we will prove: 

Proposition 2.10. Suppose 7^ y, ^- + 2, .... Then the Fourier transform 
T : 5'(M^) — > iS'(M^) induces a bijection between V-^ and V^. 

For the proof of this proposition, we prepare some general result as 
follows. Let M be a compact smooth Riemannian manifold. We write 
Am for the Laplace-Beltrami operator, Kei{^M — A) for the eigenspace 
{/ G C°°(M) : Am/ = A/}, and r for the Riemannian volume element. 
Then we can regard C°°(M) as a subspace of V{M), the space of distribu- 
tions, by /(x) f(x)dT{x). 

Lemma 2.11. Suppose A : C°°{M) — )• V\M) is a linear map satisfying the 
following two properties: 

A acts as a scalar, say a(A) G C, on each eigenspace Ker(AA/ — A). (2.14) 

a(A) is at most of polynomial growth, namely, 

there exist C, > such that |a(A)| < C{1 + |A|)^. (2.15) 

Then, A^j G C°°(M) for any ip G C°^{M). 

Remark 2.12. For a real analytic manifold M, an analogous statement holds 
for a linear map A : A{M) — )■ B{M), where A, B denote the sheaf of real 
analytic functions, (Sato's) hyperfunctions, respectively, if a(A) is at most of 
infra exponential growth (see [HI Section 2.3]). 
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Proof of Lemma \2.11\ By Sobolev's lemma, a distribution T on M belongs 
to C°°(M) if and only if A^jT (in the sense of distributions) belongs to 
L^{M) for any I G N. We will show that this is the case for T = Aip if 
G C°°(M). 

Let = Ao > Ai > A2 > . . . the (negative) eigenvalues of Am, repeated 
according to their multiplicites. We take an orthonormal basis {(pj : j = 
0,1,2,...} in L'^{M) consisting of real-valued eigenfunctions of Am with 
eigenvalues Xj. Then any distribution T on M can be expanded into a series 
of eigenfunctions ( distribution): 

j 

For / G N, the condition A^j^T G L^(M) amounts to 

^|c,f Af < 00 , 
j 

so that 

T G C^{M) ^ J2 (1 + \^]\?^ < 00 for any / G N . 

j 

Take any ijj G C°°(M), and expand it into a series of eigenfunctions 

3 

Applying the operator A, we get from fl2.14p 

All) = ^bja{Xj)ipj . 
j 

For any / G N, using ( ETT^ 

5^ IM(A,)pAf < |6/(l+|A,|)^^Af < |6/(1+|A,)P^+^' < +00 . 
j j j 

Thus Alp G C~(M). Hence Lemma I2III] is proved. □ 

We are ready to complete the proof of Proposition 12.101 
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Proof of Proposition \2.1CK For / G K,^, J-"/ is a homogeneous distribution 
of degree —jj, — y. Therefore, to see that J-"/ G C°°(M^ \ {0}), it is sufficient 
to show that the restriction J^f\sN-i is a smooth function on S^~^. This 
follows from the general result (see Lemma I2.1ip together with Lemma 12.71 
and Stirling's formula on the asymptotic behaviour of the Gamma function. 
Hence. Proposition 12.101 is proved. □ 



2.5 Operator TJi and Symplectic Fourier transform J^j 

The key idea to find eigenvalues of the integral transform 7^ on L^(S'^"^^) 
is to interpret it as the restriction of the symplectic Fourier transform, to be 
denoted by J-'j, on the ambient space M^'^. 

If Re < — Y + 1 , the following integral converges absolutely for any 
h G C°°{S^-^): 



N _ 



iQMv)--= / \{u,v)\-'^-^hiu)du. (2.16) 

Then Q^h extends meromorphically on /i G C, whose poles are simple and 
contained in the set {1 — 3 — y, 5 — y, . . . }. Thus, we get a family of 
linear operators that depend meromorphically on /i G C by 

Qf, : C°°(5^-i) ^ C°°(5^-i). 

We may regard Q^/i as an even homogeneous function on \ {0} of degree 
—fi — Y by simply letting r/ be a variable in \ {0}. Then, Q^h G V^. By 
Proposition I2.10( the Fourier transform J-" gives a bijection between V^^ and 
Vf, for ^ f , f + 2, . . . . On the other hand, Vf, can be identified with the 
space of smooth even functions on S^~^. We notice that the latter space is 
independent of fi. Thus, we have the following diagram: 

u o u 
n n 
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The lower diagram commutes up to a scalar constant. To make a precise 
statement, we define 



(2vr; 



C7v(^) :- — _ , ^ ^ ^ 



r(yU + cos + 2 y 

~ r( A^+2M -) ■ (2-17) 

Then we have: 

Proposition 2.13. ^45 operators that depend meromorphically on fi, sat- 
isfy the following identity: 

Proof. Any element in V-^ is of the form 

h^_N{ruj) = rf"^h{u) (r > 0, ue S^'^), 

for some h G C°°(5'^^^) which is an even function, i.e., h{u) = h{—u). 
We shall prove 

g^Vf = Civ(/i)-FVf (2-18) 

as distributions on \ {0}. For each fixed h, the both sides of fl2.18p are 
distributions that depend meromorphically on /i. Therefore, it is sufficient 
to prove f l2.18p for some non-empty open domain in fi, say, 

Re/i>-y. (2.19) 
The inequality f l2.19p implies that h^_N G Lj^^iM.'^), and we have 

lim e"^''^'"/i, _ N (rtu) = h, n (ru) 

as a locally integrable function, and also in Hence, taking the Fourier 

transform, we get 

£4,0 A* 2 ' 1^ 2 

in S'iR^). 
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Let us compute J^{e ^'^^^/i^_jv). Below, we use the Fourier transform for 

both and M, which will be denoted by J-^iv and J-k to avoid confusion. 
We note that e-^^^''h^_N E L^{R^) if £ > and /i satisfies (|219|) . Let s > 

and rj G S"^"^. Then the Fourier transform can be computed by the Lebesgue 
integral: 



5^-1 Jo 



oo 



SN- 

TV 

s{u,T]) - ie)~^~^h{u)d(y{uj). 



r(/i + f )e-^('^+f ) 



Taking the limit as e — )■ 0, we get 

r(;, + f)e-f('^+f) r _ 

^Riv/i ^^(sr/) = — ' ^ M — / ((w,^)-«0) ^ ^h{uj)da{u), 

where ((a;, ri)—iO)^ denotes the substitution of a; = {u, rj) into the distribution 
(x — iO)^ (see Example I2.5p . Since h is an even function, the above integral 
amounts to 

e^f(A'+f)cos^ + [ \{co,ri)\-^'-^h{tu)da{Lo) 



2 V 2 



SN- 



1 



by (12 .71) . Therefore, J^Miv/i^_iv (sr^) equals 



= Cjv(/i)-'s-'^-^(Q^/i)(r7). 
Thus, Proposition 12.131 has been proved. □ 

So far, has been an arbitrary positive integer. Suppose now that is 
an even integer, say, = 2n. We introduce the symplectic Fourier transform 
defined by the formula: 
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We identify R^" with C" by (x,^) x + i^. Correspondingly, the complex 
structure on M^" is given by the linear transform 

Then the formula f ll.ip is equivalent to 

[X,Y] = {X,JY) (X,FgR2"), 

and therefore, our J^j and the usual Fourier transform J-K2n are related by 
the formula: 

(J-j/)(F)= (2.20) 

Likewise, the linear operators 7^ (see ( I2.ip ) and for N = 2n (see 
(I2.16P ) are related by 

%fiY) = Q,{JY). 
Therefore, Proposition 12. 131 leads us to: 

Proposition 2.14. Let C^ifJ') be the constant defined in f l2.17p . Then, 

% = C'2n(/^)^J ,/ • 

Remark 2.15. Since the symplectic Fourier transform J-'j induces a bijection 
■Fj\y : V-^ ^ for all /i G C, Proposition 12.141 implies that 7^ is also 
bijective as far as C2n(At) 7^ 0, oo. 

We note that C2n(/^) has simple zeros at // + n = 0, —2, —4, .... In this 
case, the kernel y]| ^ " is a polynomial in Y of degree — (/x + n), and 
correspondingly, {T^f){Y) is also a polynomial of the same degree. Thus, 
Image 7^ is finite dimensional, and Ker 7^ is infinite dimensional. 

On the other hand, C2n(/^) has simple poles at + n = 1, 3, 5, . . . . This 
corresponds to the fact that the distribution of one variable has simple 
poles at A = —1, —3, —5, . . . (see [7]). 

We are now ready to complete the proof of Theorem 12.11 

Proof of Theorem \2.1\ Suppose p G 'H"''^(C"). Since J acts on Zj (1 < j < n) 
by and Zj by — a/— 1, we have 

p{Jr,) = (-l)^p(r7). (2.21) 

In view of Lemma [2. 7[ Proposition I2.14[ and (12. 20 p . the operator 7^ acts on 
n^'^iC") as a scalar 

(-l)^C2„(/i)52n(/i - n, a + 13). 
This amounts to {—l)'^Aa+i3{^), whence Theorem 12. II □ 
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3 Proof of Theorem 



1.1 



3.1 Dimension formulas for spherical harmonics 

This subsection summarizes some elementary results on the dimensions of 
harmonic polynomials in a way that we shall use later. They are more or less 
known, however, we give a brief account of them for the convenience of the 
reader. 

Let 'P'^(R^) be the complex vector space of homogeneous polynomials in 
variables of degree k. Its dimension is given by the binomial coefficient: 



dimV'' 



k + N-1 
k 



(3.1) 



In light of the linear bijection (see e.g. ^ pp. 17]): 

n'iM'') © V'-'iR'') ^ P'(M^), (p, q) ^ p{X) + |X|2g(X), 

we get the dimension formula of ?/*^(M'^): 

dimn^R^) = dimV\R^) - dimV''-\M^) 
_ {k + N -3y.{2k + N -2) 
~ k\{N-2)\ ■ 

In the next subsection, we shall use the following recurrence formula: 

Lemma 3.1. dim?^*^(M^) + dimH^-^(M^+^) = dim?^*^(R^+^). 

Proof. By the elementary combinatorial formula 

m\ f m \ _ fm + 1 
^k) ^ \k-l) ^ \ k 

we have 

dimP*^(M^) + dimV^~\M.^^^) = dimP'=(R^+i). (3.2) 

Taking the difference of fl3.2p for k and k — 2, and applying fl3.ip . we get 
Lemma 13.11 □ 

To find the dimension formula of 'H"'^(C") one might apply the above 
method (see e.g. [161 Section 11.2.1]), but it would be more convenient for 
our purpose to use representation theory. There is a natural action of the 
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unitary group U{n) on 'H°'''(C"). This representation is irreducible, and its 
highest weight is given by (a, 0, ... , 0, —(3) in the standard coordinates of the 
Cartan subalgebra. By the Weyl character formula, we get 



n—l 



If we use the Pochhammer symbol {a)i defined by 

a « := ' , = a a + 1 ■ ■ ■ a + / - 1 , 
T{a) 

then we may express these dimensions as 
dim'H 



_ (fc + lk-3(2A; + iV-2) 



' r(A^-i) 

dimT^-z'lC") = i- + P + --l){^ + l)n2W + 

r(n)r(n — 1) 
3.2 Alternating sum of dim'H'^'^(C") 

By the direct sum decomposition (12. 3p . the following identity is obvious: 
dimH^X^^") = dim7{"''^(C"). 

a+[S=k 

However, what we need for the proof of Theorem 11.11 is an explicit formula 
for the alternating sum: 

D{k):= J] (-I)^dim7/"'^(C"). 

a+l3=k 

Clearly, D{k) =0 for odd k because dim'H"'^(C") = dim?^^'"(C"). 

A closed formula of D{k) for even k is the main issue of this subsection, 
and we establish the following relation: 



Proposition 3.2. 



D{21) = dim?/'(M-+^) = ^ ^J'X' r (3.4) 
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Remark 3.3. The Pochhammer symbol {a)i may be regarded as a meromor- 
phic function. Thus, the right-hand side of fl3.4p can be regarded as a mero- 
morphic function of n. In this sense, the right-hand side of (13.41) still makes 
sense for n = 1. 

The rest of this subsection is devoted to the proof of Proposition 13.21 For 
this, we set 

X(^):=x' + ^ for / = 1,2,.... 
It is readily seen that X^'-^ is expressed as a monomial in 

X ■=x + - 

X 

of degree /. For example, 

X(^) = X, X(2) = - 2, = - 3X, . . . . (3.5) 
For an arbitrary /, we have the following formula: 
Lemma 3.4. 

XC) = Y^{-iy dim-H^(M'+2-2J)X'-2^'. (3.6) 
i=o 

Proof. We prove Lemma [3.41 by induction on /. The equation (13. 6p holds for 
/ = 1, 2 by (13. 5p . Suppose / > 2. We shall prove the equation (13. 6p for / + 1. 
We use 



= xx(')-x('~i). 

By substituting (13.60 for / and / — 1 into the right-hand side, we get 

[1] 

XC+i) = ^(-l)idim?^^(M^+2-2i)X^+^-2^^ 



j=0 



^(-l)Mim-H^(M'+i-2*)^«-i-2i 

i=0 



X'+i + ((-l)^(dim?{^(M'+2-2i) ^ dim-H^-i(R'+3-2i))x'+i-2i) . 
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To see the second equality for odd /, we note that dim'H'^(R^) = for d > 2, 
and thus 

dimH^ (M'+2-2i) _ for j = (3.7) 

Applying the recurrence formula given in Lemma I3.H we get (I3.6p for 
/ + 1. By induction, we have proved Lemma [3.41 □ 

Proof of Proposition \3.S[ We take a maximal torus T of U{n) and its coor- 
dinate (xi, . . . , Xn) such that 



T = {x = (xi, . . . , Xn) € C" : \xi\ 



1}, 



and that the linear map J : 



p2n 



p2n 



is represented as J 



-1,..., 



-De 



T. Then the character xh''(m.^"){9) of the representation of 0{2n) on "H ( 
takes the value 



kfTn>2n\ 



J2 (-1) 2 dim-H"'^(C") = (-1)2D(A;) 

a+l3=k 



at g = J. 

By using this observation, we shall analyze the character X'H''{r^"){9) 
g approaches to the singular point J & T. 
Let 



and we set 



:i<j<n, (eN), 



/A1 



Sk{x) := det 



(n-2) 



\ 1 



(1) 



X. 



(1) 



(fc+n-1) 



Xn 
^(n-2) 



Then, by the Weyl character formula for the group 0{2n) and by using a 
trick which reduces the summation over the Weyl group for 0{2n) to that 
over the symmetric group Sn (see [E]), we have 



Sk{X) 
So{x) 



for X G T. 
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Since X^) = mod Q-span(l, X, . . . , X' ^) an elementary property of 
the determinant shows: 



Skix) = det 



(k+n-l) 
2 

n—2 



X"-2 



V 1 



X2 
1 



Xr, 



J 



As Xj goes to Xj tends to (1 < j < n). Therefore, we have 



X?l(R2n)(<^) 



hm 



S2l[X 



Xi,...,Xjv-i>0 So[X) 

= the coefficient of X"~^ in the expansion (13. 6p 
for 

= (-l)'dim?^'(M"+i). 
Here, we have used Lemma [3.41 for the last equality. Thus, we have proved 

D{21) = dim?^'(M"+^). 
The second equality of (13. 4p is immediate from (13. ip . □ 



3.3 Triple integral as a Trace 

We are now ready to prove Theorem 11.11 As we remarked in Introduction, 
the both sides of Theorem 11.11 are meromorphic functions of Ai, A2, and A3. 
Therefore, it is sufficient to prove the identity in Theorem II. II in an open set 
of the parameters (Ai, A2, A3) G C'^. 

By the change of variables fij := |(Ai + A2 + A3 — n) — Aj (1 < j < 3), we 
first consider the case when Re/ii ^ 0, Re/i2 <^ 0, and Re /is <C 0. Then, 
the operators 7^^, 7^2 5 7^3 are Hilbert-Schmidt operators on L^(5'^"'~^). 
In particular, the composition T^iT^a'^Ms trace class, and its trace is 

given by 

Trace ( 7^ J 7^2 "^3) 

= / |[x,r]p'''""|[r,z]p'''""|[z,x]p'''""dcT(x)rf(T(r)rfa(z). 
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On the other hand, the trace of the operator T^iT^a'T^s '^^^ ^e also computed 
by its eigenvalues. Therefore, by using Theorem 12 ■![ we have 

Trace(r^,r^,r,3) = E ( n(-l)^^a+/3(/i.) ) dim?^"''^(C^) 

oo 3 / 

fc=0 j=l \a+l3=k 
oo 3 

/=0 j=l 



Applying Proposition 13. 2[ we get 

oo 

Trace(r^,r^,r^3) = E^2;(/^i)^2/(/i2)A2/(/i3)dim?/'(M"+i). 

1=0 

In light of the recurrence relation: 

A21M / + ^ 



AM 1 + 



n—fi ' 



the meromorphic function A2i{n) can be expressed in terms of Pochhammer 
symbols as 

AM = j^Aoi^), 

where 



Aoin) = 2n--r-j-^. (3.8) 



Therefore, 

Trace (7^^ 7^3 ) 

_ ]_ "+1 "+^2 n+/j3 

^(/il)^(/i2)^(/i3)5i^4 I n-M3 ' ^ 1 • 

2 2 2 2 
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Here 5F4 is a generalized hypergeometric function. 
A generalized hypergeometric function 



p-t q 



(3i ■■■ (3q 
is called well-poised (see [T]) if p = g + 1 and 

1 + «! = ^2 + /3l = • • • = ttp + 

In particular, our case is well-poised, and we can use the following Dougall- 
Ramanujan identity (see [loc. cit., pp. 25-26]): 

(m — 1 ^^Y^ —X —y —z 
X + m y + m z + m 
r(x + m)T{y + m)T{z + m)r(x + y + z + m) 
r(m)r(x + 1/ + m)r{y + z + m)r(x + -2 + m) 

Together with (13.81) . we get 

(27r"- i ) 3r ( ) r ( ) r ( ) r ( ; 



Trace (7^^ 7^2 '7^3 ) 



r(n)r(-i^)r(-^i^)r(-i^; 



(3.9) 

Now, Theorem 11.11 follows by substituting /xi = — ^(a + n), /X2 = — |(/3 + 'T'), 
and /i3 = -|(7 + ?^). 



4 Other triple integral formulas 

In this section, we discuss explicit formulas for the integrals of the triple 
product of powers of \x — y\ and \{x,y)\ instead of those of the symplectic 
form |[X,F]|. 



4.1 Triple product of powers of \x — y\ 

In this subsection we consider a family of linear operators that depend mero- 
morphically on /x G C by 
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defined by 

(^m/)(^)=/ \io-v\''-'^f{oo)da{oo). (4.1) 



The multiplier action of 7^^ on spherical harmonics is known (see e.g. [2]): 



= ^Mid, (4.2) 



where 7fc(/i) = 7A;(/i, M'""'"^) is given by 

r(m + |)r(-f)r(fc + ^ 
2/^+1 v^r(^)r(A; + 



Then, by an argument parallel to Section 13. 3[ we can obtain a closed 
formula for the triple integral built on TZ^ (see Theorem 14.21 below) . Instead 
of repeating similar computations, we pin down a comparison result between 
the two triple integral formulas by using Proposition 13.21 This comparison 
result explains the reason why the same method (e.g. Dougall-Ramanujan 
identity) is applicable, and seems interesting for its own sake. 

Proposition 4.1. 

Trace(7^^,7^^,7^^3 : L^S^ ^ L\S^)) 

= cTTace{%,%,%, : L^S'^-') ^ L\S^'^-')), (4.4) 



where 



^_,' r(m + i) yj^ r(-f 



^ j=l \ 2 

Proof. By (142|) the left-hand side of (H^ equals 

3 



n^'^('"^'^'"^') ) dim?^^(M-+i). (4.5) 

fc=0 \j=l 



Comparing (14. 3 p with Theorem 12.11 we get 

7fc(^,M-+i) _ r(m+i)r(-f) 



^2fc(/i,C") 2'^+%"^r( 



-/^— m+l ' 



(4.6) 



By (14. 6 p and (13. 9p . we see that (14. 5 p equals the right-hand side of (14. 4p . 

□ 
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The right-hand side in Proposition 14.11 was found in fl3.9p . Then, by a 
simple computation, we get 

Trace(7^/,,7^^2 7^^3) 

_ / r(m + |) \' r( -^^-^|-^3-'" ) ' r(-f) 

~ V 27r5 J r(m) H 2M,r(^Mid/iL+/fl±^i3))' 

Finally, substituting fij = i(Ai + A2 + A3 — m) — Xj (1 < j < 3), we have 
proved the following: 

Theorem 4.2. Let a,/3,7, and 6 be as in Theorem \l.l\ 

I \Y - Z\'^\Z - X\^\X -Y[^da{X)da{Y)da{Z) 

~ V 2i-f ; 2^^^r(m) r(^)r(^^)r(^) ' 

We will give in Proposition 16.91 the domain for the absolute convergence 
of the above integral. 

Remark 4.3. The formula in Theorem 14.21 was previously found by A. Deitmar 
[6] by a different method; namely it established a recurrence formula bridging 
SOo{i + 1,1) to SOo{i — 1, 1) and used the Bernstein-Reznikov formula for 
500(2, 1) and an analogous formula for SOo{3, 1). 



4.2 Triple product of powers of 

In this subsection we consider the third case, namely, the linear operators 

see fl216|) ) 



: C~(5^-i) -> C^{S^-^) defined by the kernel \{x,y)\-^' 
and the corresponding triple product integrals. 
Here is the counterpart of Theorem 12. II for Q^: 



N_ 
2 



Proposition 4.4. 



y2l 



for odd k, and 
= CN{fJ^,l) id, 



where 



CAr(/i,/) 



,2vr^r( 



4 



r(/ + 



2£i+iV^ 
4 ^ 
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Proof. By Lemma 12.71 and Proposition 12. 13^ we have 

CAr(/i, /) = Cn{^)Bn{^- - y , 21). 



As in the previous cases, we have 



Trace(Q^,Q^,Q^3 : L^S'^-') ^ L^S'^^')) 

00/3 \ 
= ^ nciv(/ij,/) dim?/^ 



i2UTaN\ 



1=0 \j=l 

By substituting 

c^(/i,0 = (-l)'c^(/i,0)^ ' ' 
dim7^2«/^iVN 



into the right-liand side of (14. 7p . we see that (14. 7p equals 

3 \ oo 3 f N+2fMj ^ ,N ( N-l \ ( N+2 \ 



\j=0 / j=0 j=l V 4 



3 /TV _ 1 N+2 N-1 N+2ni N+2/i2 N+2fi3 

2 4 2 4 4 4 



JJ Cat (/ij, 0)6^5 



jV-2 1 N-21J.2 N-2n3 

j=0 \ 4 2 4 4 4 

By using Whipple's transformation ([H p.28]): 

p fo'i 1 + |a, fe, c, d, e 

^ ^ Y |a, 1 + a — 6, 1 + a — c, 1 + a — (i, 1 + a — e 

_ r(l + a-d)r(l + a-e) 
~ r(l + a)r(l + a-d-e) 

/I + a - 6 - c, ci, e 
^ l + a-6, 1 + a- c' 



27 



we get 



Trace (Q^, 2^2 2^) 



Hence we have proved: 

Theorem 4.5. We have the following identity as a meromorphic function of 
(1^1,1^2, 1^3): 

[ I {y, z)\-^^^ I x)\-^^- 1 (a;, y) \ -'^Ha{x)da{y)da{z) 

r(f )r(-., - .3 + f )r(-., + f ) ^"^^ 1^ \ + f j ' 

We will give in Proposition 16.71 the precise region for the absolute conver- 
gence of the above integral. 

5 Perspectives from representation theory 

In this paper we have proved closed formulas for the triple integrals (see e.g. 
Theorem 11.11) . based on a combination of methods from classical harmonic 
analysis. As we have seen, these methods allow us to establish explicit for- 
mulas for symplectic groups of any rank, and even in rank one case it gives 
a new proof of the original results due to Bernstein and Reznikov [S] and 
Deitmar |i5j. 

So far we have avoided infinite dimensional representation theory, which 
was not used in our proof of main results. On the other hand, there are a 
number of interesting perspectives of these formulas, and also of the steps in 
its proof, that deserve comments. 

One aspect of Theorem 11.11 is that the triple integral considered therein 
arises from a particular series of representations vr^ of the symplectic group 
G = Sp{n, M) of rank n induced from a maximal parabolic subgroup P G G 
and depending on a complex parameter fi. Section [5] highlights this point 
mostly. 



+£f3 \-p/7V-2/ii • 




2 ^ ^ 4 



: («) 

2 4 
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Another aspect is that of analytic number theory, which was the main 
theme of [HIS]. Motivated by the classical Rankin-Selberg method, authors 
considered a cocompact discrete subgroup of the rank one symplectic group 
and automorphic functions on the associated locally symmetric space. The 
product of two such functions may be decomposed in terms of a basis of 
automorphic functions and the corresponding coefficients are related to au- 
tomorphic L-functions. The closed formula (n = 1 in Theorem II. ip gave an 
estimate of their decay [5]. 

Yet another aspect of the above mentioned triple integral is that it arises 
also in pseudo-differential analysis of the phase space M^". This phenomenon 
was treated in [13], where the symmetries of the Weyl operator calculus on 
the Hilbert space L^(M^"') were considered. 

5.1 Invariant trilinear forms 

Now we focus on some links between the triple integrals discussed in Sections 
1-4 and representation theory of semisimple Lie groups. 

We begin with a construction of an invariant trilinear form based on the 
Knapp-Stein intertwining operators. Let G be a connected real semisimple 
Lie group and P an arbitrary parabolic subgroup. Let P = MAN be a 
Langlands decomposition, a and n the Lie algebras of A and respectively, 
and 2p the sum of roots of n with respect to a. Take a Cartan involution 6 
of G stabilizing MA and set = {^f G G : 6{g) = g}. 

For A G we define (possibly degenerate) principal series representa- 
tions of G, to be denoted by nx, on the space of smooth sections for the 
G-equivariant line bundle Cx+p = G Xp Cx+p over the real fiag variety G/P, 
equivalently on the vector space 

Vr = Vx := {/ G G^iG) : figman) = a-'-^fig), ^man G P}. 

In our parametrization, is the volume bundle !\_'^^^'^/^T*{G / P) over G/P. 
Similarly, the space of distribution sections for Cx+p will be denoted by V^'^ . 
These representations are called spherical because Vx contains a i^-fixed 
vector 1a which is defined by the formula: \x{k'man) := a^^~P for kman G 
KP. 

Denote by P = MAN the opposite parabolic subgroup to P. Assume 
that it satisfies the condition: 

CI. P and P are conjugate in G. 
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Then there exists the G- intertwining operators T\ : V^x — i- V\, referred to as 
the Knapp-Stein intertwining operators [10], that depend meromorphically 
on A. They are given by the distribution- valued kernels Kx{x,y) G V^^^ ® 
V^"^ such that irxf){x) = {f{y), Kx{x,y)) E Vx for / e V.x- The Knapp- 
Stein kernel Kx may be thought of as a distribution on G x G subject to the 
following invariance condition foigEG and mjajrij G MAN {j = 1,2): 

Kxigxniiaini, gym2a2n2) = a^^'''a2^'^Kx{x,y). (5.1) 
For /, e Vx^ (j = 1,2,3), we set 

Tai,A2,A3(/i, /2, /s) 

:= (Ki(„_^)(l/,2;)iri(^„^)(z,x)iri(^_^)(x,t/),/i(x)/2(i/)/3(^)), (5.2) 

where a = Ai - A2 - A3, /3 = -Ai + A2 - A3, 7 = -Ai - A2 + A3 G a^. 
We have the following: 

Proposition 5.1. Assume P and P are conjugate in G. Then there exists 
a non-empty open region of (Ai, A2, A3) G (a^)^ for which the integral / I5.^)) 
converges. It extends as a meromorphic function o/Ai, A2 and A3. Then, the 
resulting continuous trilinear form 

-'-Ai,A2,A3 • Vx, ® Vx, ^Vx,^C (5.3) 
is invariant with respect to the diagonal action of G: 

Tai,A2,A3 {t^m{9) fl, T^\2{9) /2, VrA3(^) /3) = Tai,A2,A3(/i, /2, /s)- 

Proof. We will give in Section |5^ a sufficient condition on (Ai,A2,A3) for 
which the integral (15. 2p converges absolutely. The meromorphic continua- 
tion can be justified by the Atiyah-Bernstein-Gelfand regularization of the 
integral (15. 2p ([3], see also [7]). Parameters a,/3 and 7 are chosen in such a 
way that the integrand in (15. 2p is a section of the volume bundle of (G/P)^. 
Whence the invariance follows. □ 

The case when P is a minimal parabolic subgroup was considered in |6] 
for G = SOo{m + 1, 1). We note that in this situation P is automatically 
conjugate to P. 

Returning to our settings, we have an isomorphism of Lie algebras: 
sp(l,M) ~so(2,l) ^s[(2,M), 
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each of which is the 'bottom' of different series of Lie algebras, namely 
sp(n,]R), so(n, 1), and s[(n,]R). Bearing this in mind, we list the following 
three cases: 

Case Sp. Theorem 11.11 corresponds to the evaluation of the trilinear form 
(15 -Sp on the K-Hxed vector 1ai ® ® Ias for the following particular pair: 
G = Sp{n, M) and P = MAN a maximal parabolic subgroup such that 
M ~ Z,/2Z, X Sp{n — 1, R) and is the Heisenberg group in 2n — 1 variables. 
Notice that S'^"'~^ is a double covering of G/P. The representation space 
can be identified with V^(M^"') introduced in f l2.13p . Then the kernel of the 
operator % introduced in is K^{X,Y) = \[X,Y]\~^~''' e 1/"°° ® 1/"°° 
which gives rise to the Knapp-Stein intertwining operator. 

Case SO. Theorem 14.21 corresponds to the case where G = SOo{m + 1, 1) 
and P is a minimal parabolic subgroup. Through the identification G/P ~ 
S"^ the Knapp-Stein intertwining operator is given by 7^^ (see (14. ip ). and the 
triple integral in Theorem 14.21 corresponds to the evaluation of the trilinear 
form (15. 2 p on the i^-fixed vector. 

Case: GL. Yet another expression of the sphere S^~^ as a homogeneous 
space is given by G/P, where = GL{N,M.) and P is a maximal parabolic 
subgroup corresponding to the partition N = 1 + (A^ — 1). The operators 
introduced in (I2.16P and involved in the Theorem 14.51 can also be interpreted 
as the Knapp-Stein integrals for representations induced from P and its 
opposite parabolic P. Notice that the condition CI fails for > 2 and 
Proposition 15.11 does not apply. 

What we have found in particular is the eigenvalues of operators 7^, 
and TZn in terms of Gamma functions. The corresponding eigenspaces are 
irreducible representation spaces of the maximal compact subgroup K. In- 
deed, in all three cases the following condition holds: 

C2. The space K/ {K n M) is a multiplicity-free space, in other words, 
(ir, n M) is a Gelfand pair. 

This implies that the representation space contains an algebraic direct 
sum of pairwise inequivalent irreducible representations of K as its dense 
subspace. Therefore the action of the operators 7^ on each ii'-representation 
space is automatically a scalar multiple of the identity by Schur's lemma. 
For example in Case Sp, K ~ U{n), the corresponding restriction 7r^|^ is 

given by W'%C"), and the eigenvalues are described in Theorem 12.11 
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In Cases SO and GL the condition C2 is also satisfied. We can see this by 
a direct computation but also by the general observation that the unipotent 
radical is abelian and consequently {K^ M fl K) is a symmetric pair. 

Another feature of our settings is the following condition: 

C3. The diagonal action of G on {G/ PY admits an open orbit. 

(In fact, there is only one such an open dense orbit except the case of 
S'L(2,M), where there are two open orbits.) 

The condition C3 is connected to the upper bound of the number of 
linearly independent trilinear forms for generic Ai, A2 and A3. If this number 
equals one then such an invariant trilinear form is proportional to the one 
constructed in Proposition 15.11 under the condition CI. 

Case Sp {n > 2) is of a particular interest: the group G is of arbitrarily 
high rank, N is non-abelian, and {K, M fl K) is a non-symmetric pair. Nev- 
ertheless all the conditions CI, C2 and C3 are fulfilled. The corresponding 
trilinear form Tai,a2,A3 has recently arisen in a different context, namely in 
pseudo-differential analysis. More precisely, a new (non-perturbative) com- 
position formula based on this trilinear form is established for the Weyl 
operator calculus on L^(]R^'^) in |T3], where a slightly different notation is 
adopted: T,„,,, ^al/i, /s, /a) = j'lxl'-XrAsif^^ Z^, /s)- 

5.2 Convergence of the invariant triple integral 

This subsection provides a sufficient condition for the convergence of the 
triple integral in Proposition 15.11 

We take a) to be the set of weights of n with respect to a. The 

corresponding dominant Weyl chamber a*^ is defined by 

a*_^ := {u e a* : (y, a) > for any a E a)}. 

According to the direct sum decomposition = a* + \^^a*, we write 
A = Re A + Im A for A G a^. Then we have 

Proposition 5.2. Suppose we are in the setting of Proposition [3T71 // 

Rea,Re/3,Re7 G -p - a+, (5.4) 
then the integral f l5.2p converges absolutely. 
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Here, — p — o^lj. is a subset of a* given by —p — a*^ := {—A — p : A G a\_}. 
The rest of this subsection is devoted to the proof of Proposition 15.21 We 
will show that the integral kernel 

i^i(„_p)(?/, z)Ki(^^_p^{z, z) 

is bounded on the triple product manifold K x K x K ii the assumption fl5.4p 
is satisfied. (As we shall see in Section |6]for specific cases, the condition (15. 4p 
is not a necessary condition for the absolute convergence.) 
Consider the multiplication map 

NxMxAxN^G, {n,m,a,n) nman. 

This is a diffeomorphism into an open dense subset G' := NMAN of G (the 
open Bruhat cell). We define the projection p by 

p : G' ^ a, nme-^n ^ X. 

We set K' := G' n K . Then we have 

Lemma 5.3. K' is dense in K. Further, if u E a*^ then mfkeK'{i^, fJ'{k)) > 
— oo. 

Proof. It follows from the Iwasawa decomposition G = KAN that any el- 
ement of G' is written as g' = kan {k E K, a E A, n E N). Since G' 
contains the subgroup AN, we get k G K'. This leads us to the bijection 
G'/MAN ~ K'/M, which then is a dense subset of G/MAN ^ K/M. 
Hence, K' is dense in K. 

In order to prove the second assertion, we may assume that G is a linear 
group contained in a connected complex Lie group Gc with Lie algebra a^irC. 
Let Pc = Mc^c-^c be the complexified parabolic subgroup of P. We take a 
^-stable Cartan subalgebra t of m. Then, t) = t + a is a Cartan subalgebra 
of 0. 

We fix a positive set A+(gc7 he) of the root system such that 5|n G 
^^(fljCt) U {0} for any 5 G A+(0C)f)c)- Suppose A G fi^ is a dominant 
integral weight subject to the following condition: 

A|o = A, A|t = 0, and A lifts to a holomorphic character of Mc^c. (5.5) 

Then we get a holomorphic character, to be denoted by Ca, of Pc by 
extending trivially on A^^c. 
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Let £^ := Gc X"^ Ca be the Gc-equivariant holomorphic bundle over 
Gc/Pc associated to the holomorphic character Ca of Pq. Then, by the 
Borel-Weil theorem, the space Fa := 0{Gc/ Pq, C^) of holomorphic sections 
for £^ gives an irreducible finite dimensional representation of Gc with high- 
est weight A. 

Let fx G -Fa be the highest weight vector normalized as f\{e) = 1. Then, 
we have f\{nman) = for nman G A^cMc^c-^c- In particular, if (7 G G', 
we get 

Since f\ is a matrix coefficient, fx\K is a bounded function for any A coming 
from the above A. In light that ]R+-span{A G a* : A satisfies flS.Sp } equals 
we have proved Lemma [5l3l □ 

Let us complete the proof of Proposition I5.2[ We recall how the Knapp- 
Stein integral operator [TDJ is given in the present context. Since we have 
assumed that the parabolic subgroup P is conjugate to P, we can find w E K 
such that w~'^Nw = N. Then, we define a function Kx defined on an open 
dense subset of G x G by 

Kxigi, 92) = e<^+''''^(^"^^'"» if g^'g^w G G'. 

It follows from Lemma below that Kx is bounded on K' x K' if —{u + p) G 
a^, and in particular, defines a locally integrable function on G x G. The 
distribution kernel of the Knapp-Stein intertwining operator coincides with 
Kx{gi,g2) when A stays in this range. 

Return to the setting of Proposition 15.21 and assume the condition (15. 4p . 
Then, by Lemma [5l3l we see that Ki(^^_p^(y, z) is bounded on KxK, and like- 
wise for Ki(^i^_p-^{z, x) and y). Since the integral (15. 2 p is performed 
over the product of three copies of the compact manifold K/M (~ G/P), 
the integral (15. 2 p converges absolutely for any fj G Vx^ (j = 1,2,3). Hence, 
Proposition 15.21 has been proved. □ 

6 Convergence of the triple integrals 

In Section 15.21 we have given a sufficient condition for the absolute conver- 
gence of the invariant triple integral in the general setting. In this section, for 
the convenience of the reader, we give the precise region of the parameters for 
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which the triple integrals in our main results converge absolutely. Section EH] 
provides a basic machinery for the convergence of the integral of the product 
of complex powers under a certain regularity assumption (16.11) . This crite- 
rion gives immediately the precise region of the absolute convergence of the 
integral in Theorem 14.51 (see Proposition 16. 7p . Unfortunately, the regularity 
assumption (16.11) is fulfilled only for generic points for the triple integral in 
Theorem 11.11 This difficulty is overcome by additional local arguments in 
Section 16.21 (see Proposition 16. 8p . 

6.1 Convergence under the regularity condition 

Let M be a different iable manifold, and /i,...,/,. G C°°{M). We shall 
always assume that the zero set {p E M : fj{p) = 0} is non-empty for any 
j (1 ^ J < For each point p G M, we define a subset of the index set 
{l,...,r} by 

Hp) ■■= {j ■■ fjip) = 0}, 
and a non-negative integer by 

r{p) := dimR-span{c?/,(]9) : j G I{p)}- 

Clearly, we have 

r(p) <#/(p) (<r). 

We fix a Radon measure on M which is equivalent to the Lebesgue measure 
on coordinating neighbourhoods (i.e. having the same sets of measure zero). 
Here is a basic lemma for the convergence of the integral of \fi\^^ ■ ■ ■ Ifrl^"" 
on M. 

Lemma 6.1. Assume fi, . . . , fr G C°°{M) satisfy the following regularity 
condition: 

"^{p) = H^Hv) for any p G M. (6.1) 
Let Ai, . . . , A,. G C. Then, ■ ■ ■ |/r|''^'' is locally integrable if and only if 

Re Aj > —1 for any j (1 < j < r). (6.2) 

Remark 6.2. The local integrability does not depend on the choice of our 
measure on M. 

Here is a prototype of Lemma 16.11 
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Example 6.3. Let M = R". We fix r < n, and set fj{x) = Xj (1 < j < 
r). Then I •Xj ]^ I ' ' ' I T'\ %S locally integrable against the Lebesgue measure 
dxi ■ ■ ■ dxn if and only if Re Aj > — 1 (1 < j < r) . 

This assertion is obvious for r = 1, and the proof for general r is reduced 
to the r = 1 case. We observe that the regularity assumption (16.11) is satisfied 
because dfj = dxj (1 < j < r) are linearly independent at any point p G M". 

The proof of Lemma 16.11 is reduced to Example 16.31 as follows: 

Proof of Lemma l6J[ Fix a point p G M, and suppose I{p) = {ji, . . . ,jk}- 
Then, by the implicit function theorem, we can find different iable functions 
yk+i, ...,?/„ in a neighbourhood Vp of p such that {fj^, fj^,yk+i, ...,yn} 
forms coordinates of Vp. 

Assume fl6.2l) is satisfied. Then it follows from Example 16.31 that the 
function Hje/Cp) \fj\^^ integrable near p. Multiplying it by the continuous 
function nj^/(p) I /j I '^^ see that ■ ■ ■ |/r.|'^'' is also integrable near p. 

Conversely, assume ■ ■ ■ |/r|'^'' is locally integrable. We will show 

(16. 2p . Take a point p & M such that fi{p) = 0. By using the above mentioned 
coordinates in Vp, we can find q ^ Vp such that fi{q) = and fj{q) 7^ 
(2 < j < r). Then the integrability of \ fi\^^ ■ ■ ■ \ fr\^'' near q is equivalent to 
that of 1/1 1'^^. This implies ReAi > —1. Similarly, we get ReA^ > —1 for all 
j (1 < J < Hence, Lemma [6.11 has been proved. □ 

Next, we discuss the local integrability of the function ■ ■ ■ |/r|'^'' when 
the regularity condition (16. ip fails. The following two examples will be used 
to determine the range of parameters for which the triple product integral in 
Theorem II. II is absolutely convergent. 

Example 6.4. The function hx{x,y) := yl'^^ is locally integrable 

on if and only if Re A^ > -1 (1 < j < 3) and Re(Ai + A2 + A3) > -2. 

Example 6.5. Let p,q > and p + q > 2. Suppose Q{z) is a quadratic 
form on MP^^ of signature {p,q). Then \x\^'^\y\^^\x + y + Q{z)\^'-^ is locally 
integrable on MP'^'^'^'^ if and only if Re Aj > — 1 (1 < j < 3). 

We observe that the regularity condition r{p) = i^I{p) of Lemma EH] fails 
at the origin in both of these examples. This failure affects the condition on 
Xj for the absolute convergence of the integral in Example 16.41 but does not 
affect in Example 16.51 
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Prof of Example \6.4\ Applying Lemma EH] to \ {0}, we see that h\{x,y) 
is locally integrable on \ {0} if and only if Re \j > — 1 (1 < j < 3). 

To examine the integrability near the origin, we use the polar coordinate 
{x,y) = [r cos6,r sinO). Then we have 



hx{x, y)dxdy = r^^-^^^+^-'+^hx{cos 6, sin e)drde. (6.3) 

Since cos 6, sin 6, cos 6 — sin 6 do not vanish simultaneously and have simple 
zero, (16. 3p is integrable near the origin if and only if Re Aj > — 1 (1 < j < 3) 
and Re(Ai + A2 + A3) > —2. Therefore, Example 16.41 is proved. □ 



In order to give a proof of Example 16.51 we prepare the following: 

Claim 6.6. Let Q{z) he as in Example \ 6. 51 Then there exists a continuous 
function A{t, 6) of two variables t G M and S > such that 

A(t,(5) ~ C(5P+'?-2 ast^O 
for some positive constant c and that 

g{Q{z))dzi- ■ ■ dzp+q = I g{t)A{t,6)dt. 
\z\<s J-52 

Proof of Claim \6.6[ Without loss of generality, we may and do assume that 

Q{z) is of the standard form Q{z) = zf + ■ ■ ■ + z^ — z^^-^ — ■ ^p+q- Taking 

the double polar coordinates 

z = {roo, S7]), r, s, > 0, w G S''-\ r] G 

we have 

/ giQiz))dz = Yol{SP-^)YoliS''-^) [ g{r^ - syP'Ks'^-^drds, (6.4) 

J\z\<5 Jb+{5) 

where we set B+{6) := {(r, s) G : r > 0, s > 0, + < 6^. By the 
change of variables R := + s^, t := — s^, the right-hand side of (16. 4p 
amounts to 

g{t)A{t,5)dt, 
where A{t, 6) is defined by 

2~+^ J\t\ 
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Putting t = 0, we have 



5P+'?-2vol(5P-i)vol(5^-i) 
^'°'''= 2^«(p+,-2) ■ 

Thus, Claim ES] is shown. □ 

We are ready to complete the proof of Example I6.5[ 

Proof of Example \6.5[ The regularity condition (16. ip is fulfilled except for 
the origin. Applying Lemma [6.11 to ]R''+''+^ \ {0}, we see that jxl'^^ lyl'^^lx + 
y + Qiz)]^'-" is locally integrable on W+'^+^ \ {0} if and only if ReA^ > -1 
(1<J<3). 

What remains to prove is that this function is still integrable near the 
origin under the same assumption. By Claim 16.61 we can reduce the conver- 
gence of the integral to that of the three variables case, namely, it is sufficient 
to show that |a;|'*'^ |x + y + t\^^A(t, 6) is integrable against A{t, 6)dxdydt 
near (0, 0, 0) G M'^ for a fixed 6 > 0. Since {x, y,x + y + t} meets the regu- 
larity condition ( 16. ip . we can apply Lemma [6.11 again, and conclude that it 
is integrable if Re A-, > —1 (1 < j < 3). Therefore, the proof of Example 16.51 
has been completed. □ 

6.2 Applications to the triple integrals 

We apply Lemma 16.11 to find a condition for the convergence of the triple 
integrals in the previous sections. The first case is a direct consequence of 
Lemma 16.11 

Proposition 6.7 (see Theorem [ISD- Let M := S^-^ x S^-^ x S^-\ and 
hx{x,y,z) := \{y,z)\^^\{z,x)\^'\{x,y)\^\ 

Then / hx{x,y, z)da{x)da{y)da{z) converges if and only if ReXj > —1 

J M 
(1<J<3). 

Proof of Proposition \6. 7\ Since hx{x,y,z) is a homogeneous function of x 
(also, that of y and z), hx is integrable on M if and only if it is locally 
integrable on M := (M^ \ {0}) x (M^ \ {0}) x (M^ \ {0}). 
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We set the following functions of (x, y, z) ^ M as fi := {y, z), /2 := {z, x), 
/s := {x,y). Then we have 

dfi = {z,dy) + {y,dz), 

df2 = {z,dx) + {x,dz), 

dfs = {y,dx) + {x,dy). 

Let us verify that the fujictions /i, /2, and /s meet the regularity assumption 
(16. ip of Lemma [6. II on M. 

Suppose y, z) = 3, namely, 

{y, z) = {z, x) = (x, y) = 0. (6.5) 

We will show r(x, y, z) = 3. If not, there would exist (a, b, c) ^ (0, 0, 0) such 
that adfx + hdj^ + cdjj, = 0, namely, 

bz + cy = 0, az + cx = 0, ay + bx = 0. (6.6) 

This would contradict to (16.51) . Hence r{x,y,z) must be equal to 3. Thus 
(16. ip holds when ^I{x,y, z) = 3. Similarly, (16. ip can be verified when 
i^I{x, y,z) = 1 or 2. Therefore, the assertion of Proposition 16.71 follows 
from Lemma [6.11 □ 

In contrast with Proposition 16.71 for the triple product of complex pow- 
ers of inner products, the regularity assumption (16. ip does not hold in the 
symplectic case. The next example discusses this situation. 

Proposition 6.8 (see Theorem [H]) . Let M = ^^"-i x ^^"-i x ^^n-i^ and 
h,{X,Y,Z) := \[Y,ZnZ,XnX,Y]\'\ 



Then the triple integral / h\{X,Y, Z)da{X)da{Y)da{Z) converges ahso- 

Jm 

lutely if and only if 

Re \j> -I (1 < J < 3), Re(Ai + A2 + A3) > -2 for n = 1, 
Re \j> -I (1 < J < 3), for n>2. 

Proof of Proposition 1 6. 8\ . Since hx{X, Y, Z) is a homogeneous function of X 
(and also, that of Y and Z), h\ is integrable on M if and only if it is locally 
integrable on M := (R^" \ {0}) x (R^" \ {0}) x (M^" \ {0}). 



39 



Similarly to the way of establishing fl6.6p in the proof of Example 16. 7[ we 
see that the regularity assumption (16. ip for (X, Y, Z) E M fails if and only if 
the three vectors X,Y, Z E M^" \ {0} are proportional to each other. Hence, 
hx is locally integrable on M \ {{au, bu, cu) : u G 5^""-^, a, 6, c G M \ {0}} if 
and only if Re > -1 (1 < j < 3). 

Let us find the condition of integrability of hx near the point (X, Y, Z) = 
(aoWO) ^Qi^O) CqWo) for some Oq > 0, 6o 7^ 0, Cq 7^ 0, and Uq G S^^^. For this 
we take coordinates as 

X = aoj, 

Y = bu + xJoj + M, 
Z = cu + yJuj + V, 

where a,b,c,x,y G M, a; G S"^"-'^, and u,v e (M-span{a;, Jw})^ (~ M^""^), 
and we consider the case where a — ao,b — bo,c — cq, x, y, u, v are near the 
origin. In view of the relations [u, Ju] = —1 and {ijJ,u) = {ijJ,v) = {JijJ,u) = 
{JijJ,v) = 0, we have 

hxiX,Y,Z)dXdYdZ 

= \ — by + cx + [u,v]\^^\ay\^'^\ax\^''^a'^"'^^dadbdcda{uj)dxdydudv. (6.7) 

Since we are dealing with the local integrability for a,b,c ^ 0, the main issue 
is the local integrability against dxdydudv. 

First, suppose n = 1. Then (16. 7p is locally integrable if and only if 
\ — by + cx\^^\y\^'^\x\'^'-^ dxdy is locally integrable on for fixed 5, c 7^ 0. By 
Example 16. 4^ this is the case if and only if Re(Ai + A2 + A3) > —2 in addition 
to ReAj > -1 (1 < J < 3). 

Second, suppose n>2. Then [m, v] is a quadratic form on of signa- 

ture (2n — 2,2n — 2). By Example 16.51 \—by + cx+[u,v\\'^^\y\^'^\x\'^^dxdydudv 
is locally integrable on M x M x M2n-2 ^ ]R2n-2 ^^^^ j^g^^. > _i 

(1 < j < 3) and this estimate is locally uniform with respect to a, 6, c (7^ 0) 
and uj G 5^"^^. Hence, the right-hand side of (16. 7p is locally integrable if 
ReAj > -1 (1 < j < 3). 

Thus, the proof of Proposition 16.81 is completed. □ 

Proposition 6.9. Let M = S"" x S"" x S"" and 

hxiX,Y,Z) = \Y - Z\^'\Z - X\^''\Z -Y\^^ . 
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Then the triple integral Jj^^ hx{X,Y, Z)da{X)dcr{Y)da{Z) converges if and 
only if 

Re Xj > -m (1 < j < 3), Re (Ai + A2 + A3) > -2m . 

The proof follows the same lines as before. 
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